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ABSTRACT * ^ • 

/ In the context of the currently developing, theory- of 
procedural knowledge and cognitive skill, the task of learning 
procedures from written instructions must involve the formation of 
production rules from the information available in the text. Since 
this process has. not been explored in great depth, a study examined * 
the relation of the production rule representation to transfer of 
training. Subjects learned procedures for operating a simple piece of 
equipment by reading step-by-step instructions. By measuring the 
reading time on individual steps and the accuracy of execution of the 
procedure, the acquisition o$ individual production rules was 
tracked. Since the procedures are related, some transfer of training 
was possible from procedures learned earlier. Results show that 
-production rules, as a way to represent procedural knowledge, can 
provide' a detailed account of the learning process, and 1 that jthe 
transfer process is more similar to comprehension processes than to 
conventional practice mechanisms or Anderson's (1982) learning 
principles. (DF) - . 
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The current theory of cognitive skill describes knowledge 
of procedures in terms of a production ru 1 fy> represent at i on which 
is constructed on the basis of an initial declarative 
( propos : t i ona 1 ) representation. In these terms, learning a 
procedure from written instructions con-sists of converting the. 
prepositional content of the written material into production 
rules. Thi<= orocess was studied in a transfer of training. 
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•experiment. Subjects Earned from step,-by-step' instructions a 
series of related procedures for operaUna^a simple .device, 
with the majo.r manipulation .being the -order of 1 earn i rig the 
'procedures'. Very strbng transfer effect? were obtained, which 
could be predicted very-well by a simple model of transfer.* 
Individual production rules can be* transferred , or re-used in- 
the representation of a procedure if they appeared in a 
previously learned procedure, meaning that learning time is 
mo-stly a 'function of the number of .completely new Droduction . t 
rules that must\e acquired-. , Examination of the'time required 
to read, individual instruction steps suggests , however ¥ that 
this transfer mechanism involves processes acting on declarative 
propositional representations of the production rules. This 
means that the transfer process is" more similar to ^comprehension 
processes rather th'an conventional practice mechanisms, or 
Anderson 1 s- ( 1982) learning -principles. • 
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' ABSTRACT 



The current theory p£ cognitive skill describes Knowledge- of 
procedures itfyterms of a production rule representation yhich 
•is • constructed >on' the. basis of ah initial declarative 
{ propositional) - representation. In these .terms,* learning • a 
procedure from written instructions consists of 'converting the 
proportional content of the 'written material into production 
rules. This process was studiej^ In a transfer. of training 
experiment'. Subjects learned from step-rby-step instructions a 
•series of relateu procedures for operating a simple device, with 
t/he diajor manipulation being ' the order of learning the 
procedures. Very) strong transfer effects' were obtained, which 
coul4 £e predicted' very well by -a.' simple model of transfer. 
Individual production rules c,an be' transferred, or re-used in th.e 
representation of a procedure f if they appeared in .a previously 
learned procedure, x meaning that learning time is mcftly a 
function of 'the number of completely new production rules that 
•must be acquired. Examination of . the time required jjo read 
individual instruction steps, suggests, However, i; that this^ 
fcr.ansfer meohanisfo involves- processes acting qn 'declarative 
propositional representations of the production rules. • This 
means that the transfer process is more similar to comprehension 
processes rather than conventional practice mechanisms, or, 
- Anderson's (1982) learning principles. 
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The Acquisition. of Procedures from Text: , 
A Production-System Analysis of Transfer of Training 
' V_ David. E. • Kieras & Sufean Bovair . 

• • ' ' •**•». ' • ' 

• .'•»♦*, 

• < • ■ 

Quite often peddle .must 1,'earn ' procedures from written 
instructions. ■ In" tfle -6ontext ' o'f\ the currently developing 
theory of ipro.cedural 'knbwledge and cognitive ' skill' ( Anderson , 
198?), this task mus^t ' involve the ' formation of production rules 
from tho information available in\text. This process has not 
been systematically explored; the - results reported here jppovide 
an ' initial, characterization.' y Two general conclusions 'follow 
from this work. ' The "first is that % a production rule 
representation can provide a very precise character izatiojn of the 
relative difficulty of learning a set of related procedures. The 
second is «that apparently there are powerful comprehension-like' 
processes that operate 'very early in learning on declarative 
representations of production rules. This supplements Anderson's- 
(1982) description of the acquisition of skill, 'in that- many of 
the important .processes involved in learning. a procedure can take 
place before a pTocedural representation has t^een iorm'ed. 

• < .*- ■ . ' 

The approach .was to have subjects lejarn procedures for 
operating a simple piece of equipment by reading step-by-st'e*p 
instructions. By measuring the heading time on individual steps, 
and • Vrie accuracy of execution of the procedure, it £s possible to 
track the acquisition of individual production rules. Since the 
procedures are related, some transfer- of training is possible 
from 'procedures learned- earlier ■ The key result* is that this 
transfer is predicted very ^well from the similarities between the- 
production system representations* for the procedures. 

♦ * • ** 

%. 

• ' DESCRIPTION OF THE TA'SK 

The subjects learned series of procedures for how to operate 
a device, consisting of a simple control panel. The goal of 
operating the device, was to get a certain indicator light to 
.•'lash. Note that, this was a rote learning situation; the 
internal organization of the device was not taught to the 
subjects. Each procedure consisted of several steps, a3 

illustrated in the step-by-step instructions in Tables 1 and 2. 
Table 1 is the procedure for r,. "normal" situation in which the 
device is operating j)ro per, iy . Table 2 is the procedure for a 
"malfunction" situatibn, in which some internal component of the 
device was not operating. Depending on the nature of' the 
raalfunct ion, the device either could be made • to work by an 
alternate procedure, or could not. The final step in each 
procedure was to signal success or failure in getting the device 
r,o work. 

* ' • BEST COPY AVAILABLE 
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Table 1 

Example of a Normal procedure 



If the command is to 'do t.he MA procedure, .then do the following: 

* • • 

Step 1. Turn the. SP Switch to ON. ^ 

«*' ' 

Step 2. Set the ]ES selector to MA. . - . \ 

+• ~.> , . * ' 

Step 3 ✓'"Press the PM button, and th,en release if. 

Jtep 4. If- th« PF indicator flashes, 

then notice that the pperation .is successful. 

otep* 5. When the. PP indicator sfpfls flashing, &4t the.ES selector 
toll. » * > 

V 

Step 6.. Turn the SP switch to' OFF. . - • 

*.*•»' 
Step':/.. If the operation was successful, 
then type "3" for success. ./ 

Step 8. Procedure is finished'. . 
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Table* 2 • 
Example of a Malfunction Procedure , 



If the command- is' t*o do the-MA procedure, then do the. following; 

ot.ep- 1 . Tjurn the SP switch to Oli. ' - * <j " ' 

Step 2. Set the ES selector to MA; . 

Step 3. Press the PM button, and then release it . . , 

Step 4. If the PF indicator does not flash,- 

xthan notice -that there is a malfunction . 

• • • 

* • • 

Step 5. If the' EB indicator is on, and the MA indicator is off, 

then notice that the malfunction might be compensated 
for. ■ 1 ' • • ' 

» « . 

Step 6. Set the ES selector to -SA. . , 

Step 7. Press the PS button, and then release it. 

Step S. .If, the PF indicator does not flash, 

then notice that -the malfunction "can not be compensated 
. for. . ' • 

Step 9. Set /the ES selector to N. 

Step 10. Turn the SP .switch to OFF. ' 

Step /I 1. If the malfunction could not be 'compensated tor, ■. 
then -type "N" for not compensated." 

'Step 12. Procedure is finished. 

• \ : 
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'The Ca.ntrol-Panel D.e ylce 

* The device used in this ' experiment was the same as that used 
' in Kieras & Bovair- (19*84)., in which the . major manipulation was 
.whether subjects were taught ( .a mental model for the internal 
organization and structure o"f the device." In this experiment, 
subjects only learned the. device by rote. -The mental model is 
included here only, to explain the behavior of- the device,, and the . 
•rationale for the choice* of procedures to . be taught -to the . * 
subjects. , , . % 

The device, is a slope-front, box wi?h a simple front panel, 
shown .in 'Figure 1, consisting of four controls,; .and four ^ 
ind icator ' lights . A laboratory computer detects the positions 
of .tt/e controls and turns the indicator lights on and off.. The 
four" control^ consist of -a toggle switrh (SP), a three-position 
selector (-ESS), and two push-buttons • (PM and^PS). -The £our 
indicator lights are labeled SPI , EBI, MAI, and PPI. The labels 
h re' bashed on ithe mental, model used in Kieras and Boy air ^ (1984) « 

The behavior of- the device- • can, most easily be described 
in terras of the diagram shown in Figure 2; which was used 
in the mental model experiments. Power flows through the 
device from left to right, controlled by the switches; and also 
affected by the state of the imaginary internal components, shown 
as boxes in the fliagrara. If a component is not functioning 
correctly,' then power cannot flow ' through it and the device 
"malfunctions . There are four components, ' EB, MA,. SA, and „ 

The 3P switch i3 , the on/off switch device and the 3P 
i rid i«--tt:)r is 'the pilot, light. The other, indicators 'are status 
l l gh v.; for 'the asaoc iated . components .. Thus, if the power switch 
Ls°On, the'EB indicator Will be on if the EB component is good, 
and off if the EB component 'is bad. 1 Note that there is no 
indicator •issociated with the SA component. 

:'owe*r flows to -the Pb component .when the ESS selector is 
.:■:♦„ io either HA Cr SA, and the corresponding button is .pushed 
I* /!- ESS-HA, FS for ESS/-SA) . When the PB" component receives 
}.ower, the PF indicator flasnes four times, and then stops until 
: -in button is released and pushed again. , t Whether these 
mb: nations of "ctfnt^rol .settings will -work" depends on the 
• t. : i : : t;f the o6raponents'»i'n the obvioua. way. For e"xam»ple, if the 
''i.i a Mi, then 'the MA ind icator ■ will not be lit', and the KSJJ-HA, 
. , '.'•omb-LnHtion will not flash the PF indicator. If the J2P or the 
}■'■< bad, then the PF indicator cannot be flashed becatfse .pjaker 
.■•stino'. Vea.'h- it, no matter how the qontrola are set. 
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EB Indicator MA Indicator . 



SP Indicator 

© 

SP 
Switch ' 



• MA 
• SA 



• N 

ES Selector 



PF Indicator 

® 

/ * • 

® ® 

FM FS 
Button Button 



/ 



F i gure 1 . ■ The control panel device. 
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Figu r e 2 . The i mu 1 ated internal structure of the 
device, not knotm to subjects. 
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A I though* the v re are sixteen possible "states* that the device 
-vm- b»i in, only six of these can be distinguished in terms of the 
behavior of ' 'the device. • For example, if the EB component 
is bad,, »"then the '""status of tfre other , components is irrelevant 
because the- pattern of .indicator lights will be) the same in all 
oases. These distinguishable states are shown in Table 5, 
which shows for each state of the device What indicator lights 
wi,ll cora'e on, and which settings will make the. PF indicator 
flash. The -states are labeled by the defective component label 
prefixed with X; for example, XMA* means that the MA component is 
defective. While knowledge of the internal structure of the 
device, and component- status for each malfunction, makes it easy 
to understand the behavior of the device, it is important to note 
that this informati-on is* xxot provided to the subjects in the 
ex per iment . 



■ Table 3 

Possible States of Control Panel Device. 

v ^ 



u'tbel 



Sub-device 
'status • 



Status of 
EBI, MAI 



PFf flash on PPI flash on ^ 
'MA procedure SA procedure 



iiORHA u 

* 

xbb 



All good 

.KB bad, 
others any 



•Both on 
Both off 



yes 
no 1 



yes 
no 



XMA 



MA only bad i 
others good 



EBI on, 
MAI off 



no 



yes 



oA only bad, 
others good 

PB bad, EB 
ui.i MA good , 
.:A any 



Both- on 

Boi.i oti 



yes 



no 



^no 



no 



MA bad » i^B 
good*, PB or 
'.'A or both b-id 



KB.I on, 
MAI 'off 



no 



no 
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Ofjor-itiri^ Procedures 

Then** are a total of twelve procedures that could, "be used to 
^operate the device in it's six possible states, six where the .HA- 
setting of the ESS is tried firsthand six Iri which the SA is 
first. In the experiment, the subject was commanded, to do either, 
the MA procedure of the SA procedure , where these commands 
referred to which ESS setting was , to be tried first.' Of the 
twelve possible "«ocedures.,' the PF indicator can be made to flash 
in six (NORMAlT, XMA, and XSA) . * If was decided -that the 
def i-nitWn of ■ a malfunction should be tljat t'he -first settings 
tried would not work, and so SA-XMA and MA-XSA were not included, 
leaving a ^total "of ten procedures; two normal .and eight 
malfunction procedures. The procedure steps are listed -in Table 
4. r Tatties"! j and 2 give examples of 'the "step-^by- step, instructions 
'for a normal and a* malfunction procedure. 



Table 4 

Procedures Used to Operate Control Tanel' Device 



MA procedures 
MA-NORMAL MA-XEB 



MA-XPB 



.MA-XWA 



MA-XMA-XSA 



v1 ) SP on (1 ) SP on (1 ) SP on 

(2) ESS-MA (2) ESS-MA (2) ESS-MA 

3) FM push (3) PM push (3) EM push 

4) ESS-N (4) ESS-N (4) ESS-N 

(5) 'SP off (5) SP off (5) SP off- 

(6) Tap "S" (6) Tap "N" (6) Tap "N" 



(1 )SP on 

'.(2) ESS-MA 

C3) PM push- 

(4) ESS-SA 

L$) PS push 

(6) ESS-N 

(-7) SP off 

(8) TatD- "S" 



(1 )SP on 
(2j ESS-MA'- . 

(3) PM push/ 

(4) ESS-SA 

(5) FS push 

(6) ESS-N ' 

(7) SP off ' 

(8) Tap !'N" 
-t- 



L>A procedures 
3A-:iORMAL oA-XE 11 



SA-XMA-X3A 



SA-X3A 



SA-XPB 



{I ) 
12), 

•; 3 ) 

'. 4 ) 

lb) 



oP on 
Kl!S~ JA 
push 



E33-N 



SP (off 
Tap* "S" 



(1 ) 
(2) 
(3) 
"(4) 
5 
6 



SP on 
ESS-SA 
FS push 
ESS-N 
SP off 
Tap "N" 



(D 
(2) 
(3) 
(4) 

•ill 



SP on 
ESS-SA 
FS .push 
ESS-N 
SP off 
Tap "-N" 



(D 
(2) 
(3) 
(4) 

(5) 
(6) 
(7) 
(8) 



SP on 
ESS-SA 
FS push 
ESS-MA . 
FM pish 
ESS-N 
SP off 
Tap "S" 



(D 
2 
3 

(4) 

(5) 
(6) 
(7) 
(8) 



SP on 
ESS-SA 
FS push 
ESS-MA 
FM push 
ESS-N 
SP off 
Tap " N" 
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The eig^t malfunction procedure's can be divided into two- 
types* • -The fj.rbt % . is those in which the first ESS setting 
tried does not work, and the alternate, . setting- might work, 
depending on the. malfunction state. 'These were termed . poss i bly 
compensa t abXe . mal fun ctions * In « the second type, the alternate 
setting will not work;* and so need not be tried. These were 
"termed non-compensa t attle mal f unct ions'. For example, the XEB 
state is a non-compfrhsartable 'malfunction for .either the MA or 'the 
SA command,' and the MA-XMA and SA'-XSA statei are possibly 
compensatable malfunctions. This distinct ion^ was presented to 
tr«? subjects as part of the overall' instruct ioVs*", to rationalize 
the details. of the procedures. 




THEORETICAL ANALYSIS 

Trans fer E ffects _ 1 

In earlier work with this device (see Kieras and Bovair., " 
1983) it was noticed that the time required to learn the 
procedures under rote conditions varie J over a very wide range. 
Training t^mes for the rote-learning b ' jects are shown in Fi-gure 
y, which b,iows the training time' . r each procedure in the 
order that they were learned. Note tha^ the order of procedures 
was fixed; rather than randomized, as would traditionally be ^ 
done. Note that rather than being a smooth /descending learning 
curve, there are large peaks for the times of the third, fifth, 
and ninth procedures. The number of steps in each procedure- does 
not explain thi3 pattern, because while the number. of steps does \ 
vary for different procedures, the difference is not very large, 
and is frequently in, the wrong direction. For example, procedure 
5 has 3 steps, which is more than -procedures 3 and '4 with 7 each, 
but procedure 6 'has 9 steps. Rather, the pattern could be 
explained by the fact that the first procedure contains all new 
information, the • second (the other normal procedure) -contains 
only a little j/iew information, the third (the first malfunction 
procedure) contains some new information, the fourth (the second 
malfunot ion procedure) very little, and the fifth (the first 
pvjaiDly oompensatable malfunction) quite a lot. 
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Figure 3 . Predicted and observed training times for 
learn ing condition from IQeras and Bovair ( 1983). 
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Those data then suggest that the amount of new information 
in a procedure is -a plausible candidate for a^predictoe of 
training tame, but it is not well-defined. Transforming the 
instructions into ..production 'rules could provide a precise 
characterization of what is to be learned, thus it could be 
determined which rules could transferred, resulting in a 

quantitative measure of the amount of netf information, namely, 
the number of new production rules that must be learned. 

Product ion Rule Representation 

Table 5 pro/ides an example production rule set for the 
procedure in Table 1. The syntax of these rules is very simple. 
Each* rule is in the form: t • V 

(Label IF (condition) THEN (action)). 

The production system's working memory contains the GOALS to be 
accomplished, and NOTES, which consist 'of non-goal items 
Soncerning .current processes, the environment, or specifications 
of the tasks to be accomplished . See Kieras and Poison- (in 
press) f or * a ^ull description of the production system notation, 
along with a . description of thB user-device interaction 
simulation that was used to test the 0 production rules for 
accuracy. 

A set of production rules ,was written for each procedure 
used in. the Figure 3 experiment, and tested in the user/device 
interaction simulation to check for accuracy and completeness. 
Writing the production rules was done using a compute!* text 
editor, and it became obvious that once "the first set of rules 
was geaerated, then subsequent sets could be generated easily by 
copying the first set, doing .a few substitutions, and adding a 
f<±w rules when necessary. By analogy, -the transfer process could 
consist of recognizing which new rules are ' identical to 
previously learned rules, which are extremely similar to existing 
rules, *\nd which are totally new. The subject could then spend 
::io.it of the training time acquiring the new rules, and merely 
"tagging" existing rules as applying to the new situation. 



BEST COPY AVAILABLE 



1 1 



^ Table 5 

'Example of Production Rules 
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( ha-H-ST ART 

LP (AND (TEST-GOAL DO MA PROCEDURE) 

(NOT (TEST-GOAL DO ??? STEP))) 
THEN ( (ADD-GOAL DO SP-ON STEP) ) .) 

i 

t * 

(MA-N-SP-ON 

IF (AND (TEST-GOAL DO MA PROCEDURE) 

(TEST-GOAL DO SP-ON STEP)) 
THEN (.(OPERATE-CONTROL *SP ON) 

(WAIT-FOR-DEVICE) \ 
(DELETE-GOAL DO SP-ON STEP) 
. (ADD-GOAL DO ES-SELECT STEP) ) ) 

(WA-N-ES-SELECT 

IF (AND (TEST-GOAL DO MA PROCEDURE) 

• (TEST-GOAL DO ES-SELECT STEP)) 

THEN ( { OPERATE-CONTROL *ESS MA) 
( WAIT-FOR-DEVICE) 
(DELETE-GOAL DO ES-SELECT STEP) 
( ADD-GOAL DO FM-PUSH STEP)) ) ' 

(MA-N-FM-PUSH 

IF (AND (T'EST-GOAL DO MA PROCEDURE) 

(TEST-GOAL DO FM-PUSH STEP)) 
THEN ((OPERATE-CONTROL *FM PUSH) 
>(WAIT-FOR-DEVICE) 
(JPERATE--CONTROL *FM RELEASED) 
(DELETE-GOAL DO FM-PUSH STEP) 
(ADD-GOAL DO PFI-CHECK STEP) ) ) 

(MA-II-PFI-CHECK ; 
IF (AND (TEST-GOAL DO MA PROCEDURE) 

(TEST-GOAL DO PFI-CHECK STEP) 
(LOOK *PFI FLASHING) ) 
THEN [ ( ADD-NOTE OPERATION SUCCESSFUL) 
' DELETE-GOAL DO PFI-CHECK STEP) 
(ADD-GOAL DO»ES-H STEP) ) ) 
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Table 5 (Continued) 



(MA-II-ES-N * f ^ 

IF I AND (TEST-GOAL DO MA PROCEDURE) 
(TEST-GOAL DO ES-N STEP) 
(LOOK *PFI OFF)) 
THEN ( ( OPERATE-CONTROL *ESS N) ' % 

(WAIT-FOR-DEVICE) 
( DELETE- GOAL DO ES-N STEP) 
(ADD-GOAL DO SP-OFF ST E?) ) ) 

(MA^/-SP-OFF ' * 

LF (fylD (TEST-GOAL DO MA PROCEDURE) 

I (TEST-GOAL DO SP-OFF- ST'EP)) * 
THEN \( OPERATE-CONTROL *SP OFF) . . * 
(WAIT-FOR-DEVICE) 
(DELETE-GOAL DO SP-OFF STEP) 
. (ADD-GOAL DO TAP STEP) ) ) 

'.MA-N-7AP „ 
LP (AND (TEST-GOAL DO MA PROCEDURE) fc 
(TEST-GOAL DO TAP STEP) 
(TEST-MOTE OPERATION SUCCESSFUL) ) 
THEN ( ( DELETE- II CITE OPERATION SUCCESSFUL) 
(ADD-NOTE TYPE S-FOR SUCCESS) 
( DELETE-GOAL DO TAP STEP) 
(ADD-GOAL DO FINISH STEP) ) ) 

(MA-N-P INI SHED 

IF (AND TEST-GOAL DO MA PROCEDURE) 

(TEST-GOAL DO FINISH STEP) 

(TEST-NOTE TYPE S-FOR SUCCESS)) 

THEN ( ( DELETE-NOTE TYPE S-FOR SUCCESS) 

•/DELETE-GOAL DO FINISH STEP) 

(DELETE-GOAL DO MA PROCEDURE)) ) 
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Two, basic transfer rules were defined: identity (from 
copying), and generalization (a form of substitution). 
Production rules can be considered identical itf they have the 
same conditions' and .the same actions. The original definition 
of the generalization transfer rule was: if rules have the same 
actions, and only one point o,;f difference in their conditions, 
then the rules could be generalized. This was done by replacing 
the differing point with a, "wild-card" that mat-ches any value. 
Thus, if the only, point of difference, between two' rules was that 
one had the condition Clause (TEST-GOAL DO MA PROCEDURE), and the 
other had (TEST-GOAL DO SA PROCEDURE), then this clause could be 
replaced by (TEST-GOAL DO ??? PROCEDURE) , * Artvere "??.?" is a 
wild-card that will match' any item in that position. This 
version of the' generalization transfer rule was later modified as, 
described below. \ 

When these transfer rules were applied to -the production 
rules for the proce'dure training od^Ny* shown in Figure 3, the 
number of new rules that needed to be added for each procedure 
was determined. The , assumption is that the only rules that 
require substantial effort to learn are the completely 'new ones; 
the identical and generalizable rules should be very easy to 
learn,, since all or almost all of their content is already known. 
Thus, the number of new rules, in a procedure^ should be closely 
related to the difficulty of learning i'ne procedure. In these 
data, the number of new rules in a procedure accounts for 79# of 
the variance among the mean training times for the 10 procedures', 
supporting the value of the production system analysis of 
transfer in the learning of procedures. However, this result was 
based on only ten data points, and so is no more than suggestive. 

EXPERIMENT 

By U3ing three different training orders, this study was 
ieai^ned to get a more comprehensive set of duta on the relation 
of the production rule representation to transfer of training. 
Tne three different training orders were chosen by analyzing the 
production rule sets for each procedure using a transfer process 
simulation program, described below, and selecting training 
.orders that produced substantial variation in the number of new 
rules in each procedure, and also the number of new rules in each 
aerial position pi the training order. 



uverv i ew 

Kach subject learned a series of 10 procedures *in a fixed 
order. There were three different orders, chosen as described 
below, with a separate group of subjects for each A order. 
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Learn each procedure, the subjects first read a set 
of 3tep-by-step instructions "for the procedure,' such as those 
in Tables 1 and 2, and then attempted to execute the procedure 
on the device. If they made an error, they were immediately 
informed, an<J then began to read the instructions again. They 
were required to execute ^the procedure correctly three times 
in a row before preceding to the next procedure. * The data' 
recorded were thte reading time on each step of the instructions, 
the acpuracy of eao>Tfstep while executing the. procedure, and the 
speed "Ettui accuracy' of . a • final retention tes,t, which will not 
be discussed' in .this report. 



Method 



i 



Trans fer Simula tion. A simulation program was written in 
LISP "that could perform the transfer processes automatically. 
The program ( COMBINE-HARVEST ) can be given a set of production 
rules for a procedure which it then examines for possible 
transfer with rthe set of rules^ already known. It generates a new 
set of rules, and also reports on the number of rules considered 
identical to existing rules., the number that could be generalized 
with existing rules,, and the numb^jf of new rules jj$ded. The 
output from COMBINE-HARVEST was' tested in the user-device 
interaction simulation to - check that a proper rule set was 
generated that is, it 'followed the correct procedure in any 
given situation. 



The program's generalization criteria.wve modified slightly 
from the original definition; ' some of ' these modifications were 
iaore restrictive than the original, while- some extended the 
definition. ' The new generalization rule for transfer 
specifically excluded certain types of clauses from the 
generalization process. These are: clauses that sequence, the 
firing of rules (e.g., gofils of the form DO STEP X), clauses that 
look for a particular configuration of indicator lights on the 
device, r ,and clauses that operate controls, on the device. These 
changes in the generalization criteria mean that, in practice, 
only operations on notes, and goals can be generalized. The new 
generalization rule was extended so that it could generalize more 
than one clause in the condition, and could also generalize the. 
Equivalent clauses in the 'action part of the production rule.. 

Vhe program this would generate the number of new rules 
required for each procedure. Different orders of procedures 
produced very different patterns of the number of new rules, both 
in terras of the serial position in the sequence, and when 
•roraparing particular procedures in .different orders. Depending 
on the order in which it was processed, different training times 
would be predicted for the same procedure. 
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Training Order Conditions . Three different training orders 

were ""selected for" the experiment that, would maximize the 

predicted training time diff eren'ces. Either a procedure would 
have different predicted . training • times because there were a 
different number of new rules to be acquired, or if the number of 
new 4 rules was. the same, then the procedure would be in a 
different position in terms of the order of learning* It was not 
possible to maximize 'these differences for all the procedures, 
but it was done for as- many as possible. These different orders 
also produced different numbers of rules accepted by the transfer - 
'process as "identical ore generalized, because it seemed lively 
that recognizing identity could be a faster process than 
generalization. The training orders were therefore chosen to 
produce different values for the number of identical rules, and 
the number of generalized rules. A final constraint on the 
training orders were that they should be,, in some sense, 
meaningful orders. » . \ 

The selected training orders are shown in Tabltf 6. In 
training order condition 1, the order is: all. the MA procedures 
first, then the SA procedures:- Within this division, normal 
procedures are first, not-compensatable malfunctions second, and 
possibly compensatable malfunction last. Training order 
Condition * is based on the ?dea that once the longest procedures 
are learned, the shorter procedures should be learned 
comparatively easily. Thus,; the order for training order 
condition 2 is: .possibly compensatable malfunctions first, 
non-corapensatable malfunctions second-, and normal proceduret 
last, and within these groups MA procedures are, presented before 
The order of training order condition 3 is on the principle 
of -underlying causes, even though subjects have no information on 
these causes. Thus the ord,er is normal procedures first, then 
XKB malfunctions, then 4 XPB malfunctions, then the XMA and the XbA 
pair and- finally the two XMA-XSA malfunctions. Within these 
pairings SA procedures came before MA procedures. This order is 
quite different from the other two in that- orders 1 and 2 are 
based on the procedures themselves, actions carried out by the 
subject^ while training, order condition 3 is based cn the 
behavior of the device. 

^ Instruction Mate rials . A set of step-by-step instructions 
were '"prepared ""for """each procedure; examples appear in Tables 
1 and These were prepared so that each sentence in the 

instructions appeared to correspond to a single production 
rule, one for. each step or action (internal or external) involved 
in 'the procedure, ,and care was taken that these steps 
corresponded to the production rule sets themselves, as 
illustrated by the correspondence between Tables 1 and 5 icr the 
corresponding steps in the different procedures. 
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Table 6 0 

Number of New Production RuAs • 
for Each Training Order Condition . ;. 

Training Order Conditions 

Serial Name 'New. rules Name' New rules Name New rules 
Position ' added » added - . . added- 



1 . 


< 

MA-NORMAL 


* 

9 


MA-XMA 


13 


•SA-NORMAL 


: 9 


2. 


MA-XEB 


•5 


J4A-XMA-XSA 


4 


MA-NORMAL 


2 

*• 




MA-XPB 


1 


SA-XSA 




' SA-XEB 


5 

• 


4. 


MA-XMA-XSA 


4 . C 


* 

SA-XPB 


0' 


MA-XEB 


0 

i 


5. 


MA-XMA 


2 


MA-XEB 


1 


SA-XPB 


4 


. 


SA-NORMAL 


2 


MA-XPB 


1 


MA-XPB 


1 


.7. 


SA-XEB 


0 


j^A-XEB , 


0 


- MA-XMA 


5 


8. 


SA-XMA-XSA 


1 


SA-XMA-XSA 


1 


SA-XSA 


• 


9. 


3A-XSA 




MA-NORMAL 


2 


SA-XMA-XSA 


1 


10. 


3A-XPB 


0 


SA-NORMAL 


0 


MA-XMA-XSA 


* 0 



\ 
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app aratus . The device \ consisted of an actual physical 
control panel 'connected to a laboratory computer, which monitored 
the settings of the switches and push buttons and controlled the 
indicator lights'* accordingly. . All instructions an<V commands to 
the subjects were presented on- a standard video terminal 
positioned next . to the device. A computer-assisted instruction 
facility was used to present all of the procedure training and 
the retention teits. The subject was seated in a small room at a 
table with the^ferminal and the corftrol r~panel and was observed 
bi means of a^Video camer.a and monitor. 

Subjec ts. . Subjects, were recruited through campus, 
advertisements and were paid $5 for- their participation. 
Subjects were randomly assigned uo each of the three training 
order conditions. A total of 70- subjects participated in the 
experiment. Ten subjects' data was discarded, leaving a total of 
60 subjects, wich 2Q subjects in each condition. Of the .10 
'subjects whose data wa3 discarded,' two 'final subjects were 
discarded because their data was -not needed j three subjects did 
not finish the 'training part of the experiment , one subject was 
discarded because of a fire alarm during the experiment, and the 
first four subjects were not used because the experimental 
software required changes. , 

"D esign . Training order condition was a between-sub jects 
factof, with each subject randomly assigned to one , of the three 
training order conditions, subject to the constraint that 
approximately equal numbers were maintained in the three 
conditions. Each subject learned all 10 procedures in. all three 
.conditions. Subjects were also assigned by gender ,* so^that there 
would be an equal number of males and females .in each condition.- 

Instructions _ and . Pro cedur a. The (first pa~t of the 

instru>"tro"ns'"familiarfzed'the sub Jects. with the .layout and -labels 
on the -device. Subjects were then to'ld that they would be 
trained in several procedures for operating the device. They 
were told that t\e goal of operating the device was to make 
the PP indicator flash. .Part of their training would include 
procedures to be performed if the device malfunctioned. They 
were told that for some malfunctions the PF indicator would not 
flash at first, but it might be possible to change the control 
settings so that it would flash. This was called compensating 
for a malfunction, and it was pointed out that som6 malfunctions 
could not be compensated for. The subjects were instructed that 
whenever they were asked to turn the. device to the init ial «**tate 
that they should set the SB * switch off, the ESS selector to N, 
and hot push any buttons. 

t ' " 

The training' procedure consisted of alternating re ad J ng and 
trying phases. In the reading phase, the subject read the 
procedure " a single step at a time, in a self-paced reading 
paradigm. Then in the trying phase, the subject attempted to 
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execute the 'procedure correctly. After the attempt, the subject 
would return to the • reading phase. This process »was repeated 
until the subject had completed three correct attempts in a row. 
Then the subject would (Jommence learning the next procedure. 

, In the reading phase, the subject would tap the space bar to 
read each "step on the v terminal screen, which appeared as on« i 
sentence, as illustrated in Tables 1 and ^. The" previous step* 
was erased from the. screen. Subjects were instructed to study 
each step for as^long as they felt necessary. The lab computer 
'recorded how long** the subject left each step on the screen, 
defined as the* reading time.. V/hen the subject had read all the 
steps in the pycedur-e, a command, such as "Do the MA procedure," 
would appear on the screen and the subject would then 'tr.y to 
perform the procedure frod &em6ry. If the subject made a mis%a_ke 
while attempting >the procedure, the lab computer immediately 
sounded a buzzer, as -a signal to stop trying. Then tjie subject 
was returned to the beginning of the reading phase.. If the 
subject performed all steps correctly, the computer sounded a 
bell tone, &nd either fetv *ned to the beginning of the reading 
phase or went -on to the next procedure if the criterion had been 
achieved. Throughout the procedure, the subject was prompted by 
displays on th'e terminal screen, such as a message that they had 
:nade an error and were being returned to the reading phase. 
'< „ 

. 3'ince some pilot subjects tended to ignore the indicators 
during training," the instructions included a notice that although 
it might 3eem unnecessary to pay attention to the indicator 
lights during 'Graining, during the testing phase at the end of 
the experiment, it would be necessary to rely on the pattern of 
indicator lights to cJagose the correct procedure. 

After being trained to criterion in all 10 procedures 
subjects were instructed that they could take a short rest or 
btvak before starting the test. They were told that they 
rfoul'l see each of 'the 10 procedures three times each in the test 
in a random order. No feedback was given during testing. 



. RESULTS 

Tra L a i ng , T i me 

The total training* timp for a procedure i3 defined as 
starting when a subject begins the first reading of the first 
.sentence of the instruction steps, until completing the last step 
of the last attempted execution. 

The first analysis was simpiy to verify the presence of 
groaa effects of the the training order on training t ime • An 
•umlyois of variance was performed on the total training time for 
•vioh procedure in each training order condition; the means are 
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:)howri in Table 7. There were, inairi effects of training order 
condition and procedure, and an interaction between training 
order condition and procedure (j>s<.05). While female subjects 
<*ere an average .pf , ten seconds faster than males on 'the training, 
this difference was not significant, and there are no significant 
interactions of gender. 

' Multiple regression analyses were performed .in order to test 
' the predictions from the, theoretical analysis. The dependent 
.variable was the total training time (TRTIME) giving 600 data 
points, one for- each subject on each, procedure in each condition. 
The predictor variables were • those derived from the production 
rule model, the number* of^ew- productions (NEW), the number of 
generalized 'production rules '(GEN),_and the number of identical 
or old production rules (OLD). Other predictor variables 
included the subject's mean training time for all procedures 
(SMEAU-) to handle the within-subject design (see Pedhazur, 1982), 
Vne main elfect of training order (ORDER), and two dummy 
variables (>C0ND1 and CQND2) to test for a main effect of 
condition, with condition 3 as the baseline. Since the first 
procedure trained usually required a disproportionately long 
time, a dummy variable, FIRST , indicated whether the procedure 
was the first to be trained. Two interaction variables, C1 FIRST 
and C2FIRST, were defined to represent the interaction pf 
condition and first procedure trained. 

The results of the regression analyses on the training 
time are shown in Table 8. The table shows the coefficients 
in the final equation that includes all variables that entered 
the stepwise analysis. The F-ratios are the VF-to-remove" , and 
* so provide a test of significance of the coeTficients in the 
final equation. Finally', the standardized regression 

coefficients allow comparisons of the importance of each variable 
independently of scale differences. About 76# of the variance in 
lota:, training time was accounted for by the final equation. 

Figure 4 shows the predicted and observed mean times and the 
final regression equation. The most important predictor variable 
was the number of, new rules in each procedure (NEW), which alone 
could account for 69# of the variance, and uniquely accounts for 
about 47# of the variance. The partial and standardized 
regression coefficients for NEW are substantially -larger than 
those for identical (OLD) rules and generalizable rules (GEN), 
which are very simil&r. 

\ 
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• Table 7., 

Mean Training Times (sees)' for Each Procedure 
for the Three Training Order Conditions 



Procedure 




Training Order 


« 


Name 


Condition 1 


Condition 2 


Condition 3 


MA-NORMAL 

SA-N0RM% 

MA-XEB 

MA-XPB 

HA-XMA 

MA-XMA,XSA 

3A-XEB 

SA-XMA.XSA 

SA-XSA 

SA-XPB 


• 21 2-496 
89-863 
142.058 
109-430 
117.012 
161.478 

79.677 
86.568 

1 1 1 . 1 09 
1 1 7 . 1 09 


81.125 

92.814 . 
111.907 

96.679 ■ 
464.089 
190.291 

84-980' 

99.644 
176.169 
136.013 


111 .883 
221.958 

98.829 
108.727 
165.727 
139.679 
160.697 

95.250 
151 .41 1 
191 .817 


Mean 


122.727 


153-371 


144.598 



liable 8 
Regression Analysis 
on Total Training Time (N = 600, R2 = .7623) 



Var iaole 


Pinal 
Coeff. • 


Pinal 
Std. Coeff . 


F 


CONST AUT 


-132.39 






SMEAiJ 


1 .00 


.410 


389-78 


[JEW 


* 19-38' 


.662 


V53-54 


JuD 


1 1 .82 


.499 • 


88.44 . 


.)EN 


•1 1 .07 


.291 


51 -09 


J.-'p'iH/iT 


165. '10 


.324 


125.04 


h'IKJT 


47.10 


.155 


16.04 




-3 .-93 


-.124 / * 


18. 32 


. ' ) 1 • •/ c 


-16. 51 


-.085 


1 4.06 



X 



O OBSERVED. MEAN TIME 
• MEAN PREDICTED 4 TIME 



TRAINING TIME = -132 +~12(0LD) + 11 (GEN) + 19INEW) 
{ +47IFIRST) + 165IC2FIRST) 

-16(C2) - 4I0RDER) + SMEAN 





R 2 = .76, all-coefficients significant 
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PROCEDURE (IN ORDER TRAINED) 

Figure 4 . Mean predicted and mean observed training times for each 
training order condition. 
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In addition^ there were other** effects,' notabJLy some 
Warning- to-learn effects (FIRST and *ORDER) , and. an apparent 
"overload" effect, C2FPR3T, in which the first procedure " in the 
second training order condition took an ( extremely long amount of 
time to learn beyond that - predicted by "the number of new 
production rules. This 'procedure was MA-XMA, shown in Taftle 
4, which involved trying the MA setting first, then the SA 
setting. The other .two orders' involved only relatively simple 
normal operation steps, which may have appeared obvious and 
natural. Thus, subjects in condition 2 were confronted ' with a 
first procedure .in which the" first few steps apparently have no 
effects. 'This sprt of conceptual difficulty is clearly a 
matter for further research. 

Despite these other effects, however,, the ^production system 
variables provided by the transfer model explain the training 
times very well; in fact, the number of new rules alone accounts' 
for 69^ of the variance, and is a better predictor of training 
time on a single procedure than the subjects' individual means! 
Thus, by analyzing the procedures in terms of production rules, 
and the relations Detween them, it is possible to account 
for the difficulty of the learning the procedures with great 
precision. s 



Read ing Time * - 

The time required to read each sentence of the instructions 
was averaged over . procedures , but classified by training trial 
(e.g., first reading, second reading, and so forth), and by the 
transfer status of the corresponding production rules (Old, 
}ene rali stable >Nl ^ew) . Figure 5 shows these means. The key 
point is simple. There was a substantial .difference in the 
reading times for instruction steps depending - on the transfer 
status of the corresponding* production rule. The reading times 
for general izable and old rules were' almost identical, but 
read ing times for new rules. were much longer for the first few 
readings. A key result is that this difference appears on the 
first reading, meaning that subjects can immediately distinguish 
whether a sentence describing a step corresponds to a new rule or 
to a known one, and can immediately govern their reading and 
study time accordingly. The difference between reading times on 
trie first trial between New and Generalized is strongly 
.significant (z - 3«51» p < .01)'. 
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Figure 5 . Mean reading times for instruction sentences 
a* a function of reading trial and the. transfer status 
of the corresponding rule. 
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A second questicfa about the reading time is how thex relate 
to the acquisition of individual prbduction rules. Figure 6 
ahow3 the reading times for individual sentences -plotted in terms 
of relative t rial to mastery . The trial of mastery of a sentence 
was defined as the reading t» ial after which'the subject executed 
th*e corresponding step in the procedure correctly for all trials 
thereafter. The figure shows the mean reading times for 
sentences classified by whether the corresponding production rule 
was new, generalized, or identical. 

^ These data were subjected to a fairly complex regression 
analysis, 'Summarized in Table 9» in order to determine the 
significance of the apparant effects. In terms , of nuisance 
variables, the reading time depends on the subject's mean and the 
number of WORDS in the sentence, * and there is a simple main 
effect of relative trial number (RELTRL), corresponding, to the 
overall downward trend. There is an apparent .practice effect, 
because sentences whose S"J;eps «are mastered later., as shown by 
larger values of MASTRL, are read for less time. The key results 
are: NEW sentences are read longer than Identical or 
Generalizable, which are almost the same; . sentences before 
mastery, BEFMAS, are read longer than after; and the effect is 
mostly due to the New sentences, as .shown by the interaction 
variable BMNEW. Thus, consistent with Figure 5» sentences 
that state new rules are studied until the corresponding rules 
are mastered, whereupon they are studied for much less time. 



Table 9 
Regression Analysis on 
Individual Sentence Reading Times 
(N = 21 ,449, R 2 = .40} 



Variable 


Final - 
Coeff. 


Final • 
Std.' Coeff.. 


F 

f 


C OUST ANT 


-.557 






WOflDS 


.069 


.188 


1223.60 


oMEAH 


.844 


• 316. 


3508.06 


MAoTKL 


-.267 


-.153 . 


• 655.46 


MEW 


.723^ 


.188 


798.56 • 




-.182 


-.132 


224.06 


BEFMA3 " 


.759 


,210 


599-37 ' 


BMNEW 


1 .247 


-.208 


831 .06 
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Figure 6 . Mean reading times for instruction 
sentences as a function of relative trial of 
mastery and transfer status* 
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CONCLUSIONS 

A basic conclusion . is that production rule*, as a way to 
represent procedural knowledge, can provide a detailed account 
of important learning,, processes. ""This supports t^e approach 
suggested by Kieras and Poison (in. press; who suggest that the, 
production-rule theory of skill acquisition is useful for" 
practical applications. That* there are other phenomena involved, 
such as the "overload" described above, is. clarified by the 
production system analysis as well. 

. * 

These results present a puzzle , for the theory of skill 
acquisition as formulated by .Anderson (1982). The transfer 
process, defined here has many similarities to some" of Anderson'.s 
compilation and tuning processes. However, *hig processes 
are defined in terms 4 of operations on .procedural representations . 
These are constructed as a by-product of *the activity of general 
interpretive procedures that are driven by an initial declarative 
encoding. However, in these results, rules are 'being compared, 
modified, and constructed -very *.,rapidly, andapjpajrently before 
they exist in a procedural foritk.- As^-Flgure 5 shows, a 
generalization process can apparently -eectfr on the first reading, 
and is almost as fast as recognizing an identical rule. Although 
.there is no rigorous basis at* this time for saying so, it seems 
that these aspects of the results are not reasonably subsumed 
under Anderson's compilation and tuning processes. 

Instead, perhaps the work of relating new and old rules 
is done by processes similar -to thostf proposed for 
ma^roprocessing in "comprehension (e.g., Kiera's, 1 982), which 
iVinX compare , modify, and construct %coraplex p*ropositional 
representations while reading is going on. Thus, subjects 
translate the instruction sentence into a declarative 
representation of a complete production rule, which can then be 
related to other such representations./ If an identical 
representation already exists, it can be re-used, as appears to 
be possible in other types of text (see Johnson & Kieras, 1983). 
Similar to. Anderson's proposals, this declarative representation 
would be interpreted by a general procedure for following 
instructions, and the procedural form of the rules would 
eventually be formed by the compilation and tuning processes, 
'jawever, correct execution of the procedure' would begin when tfhe 
declarative rule set has been successfully constructed, and the. 
time 'required to do so would depend on how much use could be made 
of previously learned rule representations. Thus, when 

procedures are acquired,. from text, comprehension-like processes 
can play a major role early in learning, leaving the/ compilation 
and tuning .processes to govern learning once \he initial 
declarative form of the rules Is in place. 
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